Skip to content

storage: report actual disk usage per VM and image - #21

Merged
aljoscha merged 5 commits into
mainfrom
storage-usage-accounting
Aug 18, 2026
Merged

storage: report actual disk usage per VM and image#21
aljoscha merged 5 commits into
mainfrom
storage-usage-accounting

Conversation

@aljoscha

Copy link
Copy Markdown
Owner

Ember reported provisioned sizes everywhere and actual sizes nowhere. ember vm list printed disk_size_gib, ember info printed no capacity at all, and the one command that tried, ember debug storage-efficiency, walked .img paths that only exist on macOS and reported zero on Linux.

This adds a usage() method to StorageBackend and wires four commands to it. Design doc: docs/STORAGE-USAGE-SPEC.md.

$ ember storage usage

Pool          481.4 GiB capacity, 297.5 GiB used (62%), 183.9 GiB free
Compression   599.3 GiB logical -> 284.3 GiB on disk (2.11x)
Reserved      13.2 GiB charged to the pool but holding no data

VMS
NAME                PROVISIONED REFERENCED EXCLUSIVE  SHARED COMPRESSION
aj-dev                  200 GiB   98.5 GiB  97.2 GiB 1.3 GiB       1.97x
mz-dev                  200 GiB   10.1 GiB   8.2 GiB 1.9 GiB       2.08x
mz-dev-auto-scaling     200 GiB   92.5 GiB  89.7 GiB 2.8 GiB       2.13x
mz-dev-bugs             200 GiB   88.7 GiB  85.3 GiB 3.4 GiB       2.22x

IMAGES
NAME           PROVISIONED REFERENCED EXCLUSIVE SHARED COMPRESSION
ubuntu-dev         6.3 GiB    1.9 GiB   1.9 GiB    0 B       2.24x
ubuntu-dev-new     6.7 GiB      2 GiB     2 GiB    0 B       2.26x

The model

Occupancy, not reclaim. It answers where space went, not what a delete gives back, because on ZFS those are genuinely different questions and one field cannot honestly answer both.

Four numbers per volume: provisioned, exclusive, referenced, logical. The last two are optional, since not every backend can measure them. Shared bytes and the compression ratio are derived rather than stored so they cannot disagree with their inputs. The invariant that keeps the table readable is exclusive <= referenced.

usage() takes the state records and returns the whole installation in one value, so a backend that must walk pool-wide metadata does it once instead of once per volume.

Backends

  • ZFS answers from two zfs calls. exclusive is usedbydataset, deliberately not used: a zvol from zfs create -V carries a refreservation for its full virtual size, so image volumes report a used of 8.4 GiB against a referenced of 1.9 GiB.
  • dm-thin reads per-volume figures through a reserved metadata snapshot and thin_ls, which also covers volumes that are not currently activated. The reservation is a single slot per pool, so it is released by an RAII guard.
  • APFS reports st_blocks, leaving referenced and logical unknown.

CLI

ember storage usage is strict: being unable to measure is an error, since that is the one thing it exists to do. vm list (new USED column), vm inspect, and ember info treat usage as best-effort and render -, because one common reason to run them is that storage is broken.

ember debug storage-efficiency is removed, along with the now-empty debug subcommand tree.

Review

Two adversarial review passes ran over the first commit; the second commit is the triage. The findings worth naming:

  • exclusive was usedbydataset + usedbysnapshots, and snapshot-only space is by definition outside referenced, so both image rows shipped with occupancy above what they reference. The guard test for that invariant had been fed a fixture with the offending field zeroed.
  • The pool compression ratio counted empty refreservation as perfectly compressed data, understating it by 5% (2.01x against a true 2.11x). PoolUsage gained reserved and the ratio now divides by the occupied remainder.
  • dm-thin usage() called ensure_pool_active, so ember vm list would load a pool table, attach loop devices, and run thin_check as a side effect of listing VMs.
  • try_usage could panic on a config naming an unimplemented backend, which is exactly the config you would run those commands to diagnose. It now goes through a fallible try_create_storage.

Four findings were rejected with justification, mostly around partial results and pre-existing behavior.

Testing

cargo fmt, cargo clippy --all-targets, and 138 unit tests are clean, and the report has been run against a live 481 GiB ZFS pool.

Integration tests are in tests/storage_usage.rs, all #[ignore]d to match the rest of tests/. The macOS/APFS path is unverified — the ember-macos crate cannot compile on Linux (pre-existing, nix::libc::clonefile is macOS-only). It type-checks with that one call stubbed, but nothing about APFS st_blocks semantics has been exercised.

🤖 Generated with Claude Code

Ember reported provisioned sizes everywhere and actual sizes nowhere.
The only command that tried, `ember debug storage-efficiency`, walked
`.img` paths that exist only on macOS and reported zero on Linux.

Add a `usage()` method to `StorageBackend` returning provisioned,
exclusive, referenced, and logical bytes per volume plus pool totals.
The method takes the state records and returns the whole set at once,
so a backend that has to walk pool-wide metadata does it once rather
than once per volume.

ZFS answers from two `zfs` calls. dm-thin reads per-volume figures
through a reserved metadata snapshot and `thin_ls`, which also covers
volumes that are not currently activated. APFS reports `st_blocks`.

Wire it to a new `ember storage usage`, a USED column on `vm list`,
usage rows on `vm inspect`, and a pool line on `ember info`. Everything
but `storage usage` treats measurement as best-effort, so listing VMs
keeps working when the pool is unreachable.

Removes `ember debug storage-efficiency` and the now-empty `debug`
subcommand tree.

See docs/STORAGE-USAGE-SPEC.md.
Follow-up to the accounting change, from two adversarial review passes.

Semantics. `exclusive` was `usedbydataset + usedbysnapshots`, but
snapshot-only space is by definition outside `referenced`, so both
image rows shipped reporting more occupancy than they reference. It is
now `usedbydataset`, a subset of `referenced` by ZFS's own definition,
and the contract says plainly that this is occupancy and not what a
destroy frees. The regression test now runs against the four rows a
live pool produces rather than a fixture that zeroed the field which
broke the invariant.

Pool compression counted empty refreservation as perfectly compressed
data, understating the ratio by 5%. `PoolUsage` gains `reserved`, the
ratio divides by the occupied remainder, and the CLI prints the
reservation as its own row so the volume rows visibly sum to the pool
line.

Read-only queries. dm-thin `usage()` no longer activates the pool, so
`ember vm list` cannot load a pool table, attach loop devices, and run
`thin_check` as a side effect of listing VMs. `try_usage` moved to
`src/backend.rs`, breaking a `cli::vm` <-> `cli::storage` import cycle,
and goes through a new fallible `try_create_storage` so a config naming
an unimplemented backend no longer panics the three commands you would
run to diagnose it.

Labelling. `RATIO` is `COMPRESSION`, `Ratio:` is `Compression:`, the
accessors are `compression_ratio()`, and `ember info` says `Capacity:`
rather than a second row called `Pool`.

Tests. All integration tests are `#[ignore]`d to match the rest of
`tests/`, since the runner passes `--ignored` and `TestEnv` needs root.
Fork creation was missing `--no-start` against a kernel that cannot
boot. The unmeasurable-storage test is Linux-only, because the APFS
backend reads neither field it clobbers. Adds the unit tests the spec
asked for: derived quantities and their zero guards, metadata block
scaling, and the thin-id join.

Retargets the docs, README, and `tests/macos_storage.rs` that still
referenced `ember debug storage-efficiency`.
The APFS usage model rested on st_blocks counting only the blocks a
file does not share with a clone. It does not. st_blocks counts the
blocks a file maps, so a fresh clone reports its origin's full figure
while costing nothing, and summing it over an install of one image and
fifteen clones overstates real occupancy by more than 5x.

Replace it with a physical extent scan. fcntl(F_LOG2PHYS_EXT) gives
the byte ranges a file maps, so sweeping every .img in the tree at once
yields exclusive, shared and a true pool figure. That restores all four
columns on macOS rather than blanking them, and it keeps the invariant
exclusive <= referenced true by construction.

Also record why the trait batches: on APFS a volume's exclusive figure
is only defined relative to every other volume that might share its
blocks, so a per-volume call could not express the answer at all.

Correct the same claim where it is repeated in MACOS-SPEC.md, and flag
it as unverified for reflinks in BTRFS-SPEC.md.
The APFS backend reported st_blocks as the exclusive figure. st_blocks
counts the blocks a file maps, not the ones it owns, so a fresh clone
reported its origin's full figure while costing nothing. A pure fork
that consumed 13 MiB was reported as 285 MiB, eight of them as 2283 MiB
against 10 MiB real, and pool.allocated grew every time a free clone was
made, which also made capacity (allocated + available) drift upward on a
volume that had not changed size.

Read physical extents instead. fcntl(F_LOG2PHYS_EXT) gives the byte
ranges a file maps and SEEK_DATA skips holes, so sweeping every .img in
the installation at once splits the tree into what each volume holds
alone and what the whole set costs. Measured against a tree of one
image, two VMs and three forks: 577638400 bytes reported against
578457600 actually freed by deleting it, an error of 0.14% where the old
figure was 2.64x over. The scan takes about 30 ms.

The sweep runs over the whole tree even when the caller passes one
record. Exclusivity is not a property a volume has on its own, so a scan
narrowed to the requested records would call every volume fully
exclusive.

All four columns now carry real numbers on macOS, and
exclusive <= referenced holds by construction rather than by arithmetic,
since the exclusive bytes are a subset of the file's own extents.

The sweep is pure interval logic and is unit tested without APFS
underneath. fork_shares_blocks_with_origin stops skipping on macOS,
where it had been returning early on a null referenced, leaving the one
assertion aimed at sharing unrun on the backend whose sharing was
broken.
`default-members = ["."]` means a bare `cargo test` covers only the
root package. CI ran 30 of 321 unit tests, so every test in ember-core
and in the platform backend was green by omission, including the
thin_ls and `zfs list` parsing tests this branch added and the spec
lists as its no-root coverage.

`--workspace` is not the fix. It selects the other platform's backend
too, and that backend does not build here: ember-macos needs
`clonefile` on Linux, and on macOS ember-linux's image tests want
`mkfs.ext4`. Naming the packages per platform is what actually works,
matching how `build` already branches on UNAME.

`check` and `clippy` get the same selection, so test-module code is
compiled and linted too. That immediately caught an
`items_after_test_module` failure in this branch's own accounting
tests, now moved to the end of the file.

`build` and `release` stay as they are: they build the root package,
which pulls the platform crate and ember-core in as dependencies, so
naming them would compile nothing new.

CI calls `make test` rather than `cargo test`, so the lint, check, and
test entry points stay consistent.
@aljoscha
aljoscha force-pushed the storage-usage-accounting branch from 1173a48 to 6322dd8 Compare August 18, 2026 10:03
@aljoscha
aljoscha merged commit 086f3ed into main Aug 18, 2026
2 checks passed
@aljoscha
aljoscha deleted the storage-usage-accounting branch August 18, 2026 10:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant